Covert Aspects of Surveillance and the Ethical Issues They Raise

Analogue Surveillance in a Digital World: Covert Surveillance by Human Beings

Technological surveillance might have captured the contemporary popular imagination but surveillance by human beings – the oldest form of surveillance – is still with us. Although endemic in totalitarian societies, human surveillance by the state (as opposed to surveillance by private investigators or security companies) operates at a much smaller scale in Western liberal democracies: in the UK in 2019, there were 3,652 authorisations of ‘Covert Human Intelligence Sources’ (i.e. informants or undercover officers) and 8,049 authorisations for ‘directed surveillance’ (i.e. covert surveillance of a person in public by undercover surveillance teams). In contrast, technological surveillance is much more prevalent – for example, in the same period, there were 116,171 authorisations¹ for the use of communications data by the Metropolitan Police Service Central Intelligence Unit (Investigatory Powers Commissioner’s Office, 2020). Yet, although technology-mediated surveillance is much more common, human surveillance can intrude into people’s personal lives in a much more invasive and potentially harmful manner since it often involves deceptive relationships and betraying others’ trust.

This case study concerns the Special Demonstration Squad (SDS), which ran from 1968–2008, and similar units like the National Public Order Intelligence Unit (NPOIU²), established in 1999. In 2015, following a series of revelations and official inquiries, the UK government established a judge-led investigation – the Undercover Policing Inquiry (UCPI³) – which began to hear evidence in November 2020.

Issues Raised During the UCPI

The SDS was established in 1968 within the London Metropolitan Police’s Special Branch,⁴ the policing body responsible for national security and terrorism and liaising with the Security Service (MI5). SDS officers often adopted cover identities by assuming the name of a real person who had died as a child.⁵ They would change their appearance to blend in with activists, had vehicles and apartments in their cover names and would attend meetings, befriend activists and live their lives using these cover identities. In contrast to undercover police investigating crimes, these infiltrations were unusual in that they often lasted for several years, and officers did not collect evidence for criminal prosecutions. Over 1,000 groups were infiltrated (Evans, 2017). The information collected was quite intrusive. For example, the UCPI heard evidence that undercover officers:

recorded the political activities of children as well as details of their parents’ domestic lives. On one occasion, an undercover officer sent back to his supervisors the babysitting rota that had been organised by leftwing campaigners …. They also recorded the births of campaigners’ children and made comments in their reports about the lives of politically active parents, such as the fact they had a child with Down’s syndrome …. Some reports recorded deeply personal information, such as leftwing activists experiencing mental illness and depression or having an abortion …. Others recorded the sexuality of activists …. The police spies regularly reported on the bank accounts and jobs of campaigners, along with their home addresses. (Evans, 2021c, 12 May)

SDS officer ‘Paul Gray’ (HN126,⁶ 1977–1982) reported extensively on young people, including children active in Hackney School Kids Against the Nazis, as well as their teachers, sending photographs to his managers (Heaven, 2021). Evans (2020a, 28 October) reports that undercover officers collected information on campaigns about police injustice (e.g. the Stephen Lawrence campaign), caused miscarriages of justice because their presence was withheld from lawyers defending activists (in 26 cases officers had been arrested along with activists) and appear to have shared data with private companies enabling them to ‘blacklist’ applicants for jobs (see also Lubbers, 2015).

In contrast with technical surveillance, human intelligence can give insight into the plans and intentions of targets, but it generates many ethical dilemmas. In one of the earliest discussions of the use of human intelligence sources within social movements, Marx (1974) observed that they faced a dilemma: staying on the fringes of the group gathering intelligence passively meant they had much less access to information than if they took on leadership roles. The UCPI has heard how undercover officers often rose to key administrative positions in the groups they infiltrated, passing membership lists to their headquarters who, in turn often passed them onto ‘Box 500’, the Security Service (MI5). However, as Marx (1974) notes, being a more active and senior member of a group increases the risk that officers significantly affect the direction of the group’s activities and raises serious questions about whether they might be viewed as agent provocateurs. Marx (1974) also observes that the importance of the group and the threat is vulnerable to exaggeration:

Further, wishful thinking, limited exposure, and selective perception may lead the agent to believe a group’s own exaggerated estimates of its power and appeal and to confuse vague revolutionary rhetoric with specific plans. (p. 420)

Undercover Surveillance and Human Rights

A key concern for the activists surveilled by these officers is that their human rights were violated. Kate Wilson, an environmental activist who lived with ‘Mark Stone’ (in reality, NPOIU undercover officer Mark Kennedy) was deceived into a sexual relationship by this undercover police officer (Evans, 2021a, 20 April). In a landmark judgement in 2021 the Investigatory Powers Tribunal⁷ upheld Wilson’s complaint that several articles of the European Convention on Human Rights had been breached in her case (Wilson, 2021; Wilson v (1) Commissioner of Police of The Metropolis (2) National Police Chiefs’ Council (2021, 30 September)). The articles breached were:

• Article 3: which prohibits torture and ‘inhuman or degrading treatment or punishment’.

• Article 8: the right to respect for one’s private and family life.

• Article 9: the right to freedom of thought, conscience and religion.

• Article 10: the right to freedom of expression.

• Article 11: the right to freedom of assembly and association.

• Article 14: Protection from discrimination

These articles are also relevant to many of the other activists surveilled by these undercover units. Some of the most serious ethical issues concern the way in which many officers invaded activists’ personal lives. The UCPI is investigating the work of at least 139 undercover officers from the SDS, NPOIU and other units of whom more than 20 (i.e. over 14%) had sexual relationships with members of the groups under their cover identities (Evans, 2020a, 28 October). Many had long-term intimate relationships with activists, living with them for long periods and four fathered children with activists (Evans, 2021b, 22 April). Undercover officers often feigned mental health problems at the end of their deployment as part of an exit strategy. In 1987, animal activist ‘Bob Robinson’ suddenly broke contact with a female activist with whom he had had a child two years earlier. In 2012, she discovered that he was an undercover SDS officer called Bob Lambert. She subsequently received an apology and £425,000 compensation in 2015 after taking legal action alleging assault, negligence, deceit and misconduct by senior officers (Kelly & Casciani, 2014). She said that it was ‘like being raped by the state. We feel that we were sexually abused because none of us gave consent’ (Lewis, Evans, & Pollak, 2013).

Seven women sued the Metropolitan Police for the emotional trauma caused by such deceptive intimate relationships (some lasting up to nine years) and the subsequent apology from Martin Hewitt, an assistant commissioner at the Metropolitan Police acknowledged that their human rights had been violated:

some officers, acting undercover whilst seeking to infiltrate protest groups, entered into long-term intimate sexual relationships with women which were abusive, deceitful, manipulative and wrong … these relationships were a violation of the women’s human rights, an abuse of police power and caused significant trauma … relationships like these should never have happened. They were wrong and were a gross violation of personal dignity and integrity. (Evans, 2015)

Other Harms of Undercover Surveillance

Undercover work is a common policing tactic when investigating serious organised crime networks and these deployments are recognised as stressful for officers (Curran, 2021) and extreme levels of compartmentalised secrecy mean their families often cannot be told what they are doing and they live with a constant worry about getting ‘burned’ (exposed) or losing a target (Loftus, Goold, & Mac Giollabhuí, 2016). For those infiltrating activist groups, there are unique challenges for officers – for example, many report being violently assaulted on demonstrations by uniformed police unaware that they were working undercover (see also Marx, 1974). In some cases, officers may experience mental health breakdowns (Casciani, 2015). Their families can experience other harms and three ex-wives of SDS officers said that the deployments had caused their marriages to break down (Evans, 2020b, 4 November). At least 42 dead children’s identities were stolen and the police have admitted this caused their families ‘hurt and offence’ (Evans, 2016).

Infiltration also harms the groups targeted. Marx (1974, p. 428) notes that the discovery of informers or undercover officers can leave groups with ‘feelings of demoralization, helplessness, cynicism and immobilizing paranoia, and can serve to disintegrate a movement’. Stephens Griffin’s (2020) interviews with activists who had been surveilled revealed that their conceptions of a fixed and stable external reality were fundamentally challenged – as one participant put it ‘everyone was questioning everything’ (p. 8) with some being diverted from environmental activism. SDS ‘Officer A’ told a reporter ‘[i]f the SDS had been in existence at the time of the Suffragettes, their campaigns would never have got off the ground and they would have been quickly forgotten’ (Thompson, 2020).

There are important questions then, about balancing the state’s obligation to preserve public order and its obligation to protect legitimate debate and to provide the basis for a functioning civil society. How do we weigh up the potential harms of undercover surveillance with its possible benefits? Macnish (2015) has argued that proportionality and the level of intrusiveness are important considerations.

The Proportionality of Political Intelligence Gathering by Undercover Police

Given that the work of units like the SDS involved considerable intrusion into the personal lives of some activists, was it justified by the level of threat and were there no realistic alternatives? The SDS was formed because of rising concern about a wave of protests about the Vietnam War. For example, in a March 1968 demonstration thousands of people marched on the US embassy in London, the police lost control and more than 200 people were arrested (Evans & Lewis, 2013). The police and government were concerned about the threat of revolution posed by those it saw as subversive, especially those in anarchist and Trotskyist groups. In 1975, the Home Office Minister, Lord Harris of Greenwich, defined subversive activities as those ‘which threaten the safety or well-being of the state and which are intended to overthrow or undermine parliamentary democracy by political, industrial or violent means’ (Security Service, n.d.).

However, over the 40 years of its existence, there does not appear to have been any detailed and regular threat assessment conducted by the SDS nor any systematic consideration of potential harms and benefits. Moreover, there is evidence of inequity in the types of groups targeted. Although the UCPI have not published a full list of groups targeted, the Guardian journalist Rob Evans and the Undercover Research Group have collated a list of 135 organisations (Evans, 2019). As studies in the USA have found (Marx, 1974), the groups targeted were, overwhelmingly, on the political left, suggesting that target selection was inequitable, a key issue in deciding on the ethics of surveillance (Macnish, 2015). Out of the 135 groups infiltrated, only two were right-wing: the British National Party was infiltrated by three SDS officers and Combat-18 by one (Evans, 2019). The National Front, a violent racist group which was very active in the 1970s, does not appear to have been targeted at all.

According to evidence given by HN329, a founding member of the SDS, the unit focussed on ‘people who were opposed to the current political situation, or the current government’ (Casciani, 2020, 12 November) which seems a broader definition of subversion than that of Lord Harris and appears to simply involve opposition to the government of the day. HN329 went on to say ‘[i]t may well be that a particular group is completely harmless but we would be asked to find out what their objectives were. A file would then be opened’ (Casciani, 2020, 12 November).

However, evidence heard by the UCPI suggests that surveillance of many groups deemed not to pose a threat in terms of serious crime or violence continued. For example, the Anti-apartheid Movement (AAM) was infiltrated by four SDS officers. In an earlier Freedom of Information Act investigation by the BBC, the Metropolitan Police was found to have gathered, between 1969 and 1995, 30 inch-thick files on the AAM. These files included ‘reports of demonstrations and pickets’ consisting of ‘methodical listings of the banners carried and slogans chanted’ but ‘the documents seen by the BBC contain no evidence of the movement having been involved in anything criminal’ (Rosenbaum, 2005). Anti-apartheid activist and ex-government minister Peter Hain argued that ‘[t]he police, in targeting us, were putting themselves on the wrong side of history’ and he asked why they were not ‘targeting the agents of apartheid bombing and killing’ (Casciani, 2021a, 30 April). When an SDS officer was asked whether anti-apartheid groups had sought to overthrow democracy, he stated that ‘[i]t was not all about overthrowing democracy but nuisance – they caused problems and dangers to the public’ (Casciani, 2021b, 4 May).

The Socialist Workers Party (SWP) appears to have been a particular focus for the SDS. Although most groups seem to have been infiltrated by one or two SDS officers, the SWP was infiltrated by no fewer than 26 officers between 1970 and 2007 (Evans, 2019). For a small organisation with a membership in the low thousands with relatively little national influence, the proportionality of such surveillance seems questionable. Moreover, the SWP was already under heavy surveillance by the Security Service – Hollingsworth and Fielding (2003) report that MI5 had 25 informers in the organisation over a 30-year period whilst all 12 of its telephone lines were tapped. This level of infiltration of small groups has been seen in the USA too. Garrow (1988) reports that 17% of the Communist Party USA were FBI informants as were 11% of the US SWP even though it only had 480 members.

But, as the barrister for activists argued at the UCPI, the SDS did not appear to conduct any ‘regular and thorough risk and threat assessments which fully set out and consider any alleged risk to the public and the state from both public disorder and subversion’ (Heaven, 2021, p. 4). Indeed, a witness from the Security Service (MI5) was sanguine about the threat of subversion, noting that:

It appears … that the Security Service did not consider that subversive organisations posed a particularly high priority threat, and the pressure to investigate these organisations often came from the Prime Minister and Whitehall. (Witness Z, 2021)

However, despite this, the UCPI has heard evidence that the Security Service continued to task SDS officers with gathering information for its files.

‘Domestic Extremists’ as the New ‘Subversives’

In the twenty-first century the term ‘subversive’ has gone out of fashion and, instead, policy documents now refer to the similarly ambiguous term ‘domestic extremism’. Schlembach (2018) observes that some definitions emphasise the risk of violence whilst others refer to ‘serious criminal activity’, a much broader category if it includes damage to property and public disruption rather than violence towards people. The ambiguity and apparently widespread use of this term is likely to lead to the kinds of problems seen with the similarly ambiguous term ‘subversive’.

It is hard to escape the conclusion that these undercover units were used less because of the level of threat posed by activist groups but for pragmatic reasons like the fear of political embarrassment when they cause public disruption or the ease of surveillance by human rather than technological means. Garrow (1988, p. 9), for example, has argued that the FBI made extensive use of informants in political groups because human sources were more efficient than electronic surveillance ‘which consumed vast quantities of agent and clerical staff time while gathering, vacuum-cleaner style, far more chaff and trivia than even the FBI wanted’. Even if we accept that the state has a legitimate interest in surveilling such organisations, it is hard to believe that, in the era of big data and the extensive use of social media by campaign groups, there are not alternative, less intrusive forms of surveillance which would avoid the risk of the kinds of abuses investigated by the UCPI.

We do not know whether there is now more rigorous assessment of the threat posed by groups and whether there are mechanisms for weighing up the potential harms and benefits of such intrusive surveillance. Evidence given to the UCPI suggests there was previously a level of disregard for the range of potential harms which bordered on the reckless. Although there might be a temptation to regard abuses as the result of a small number of ‘rotten apples’, the number of officers involved suggests that the failings are of a systemic nature. Given that the vast majority of undercover officers involved in sexual relationships with activists were men (as were their managers), some activists have argued that the apparent lack of guidance about sexual relationships with activists indicates the existence of institutional sexism (Evans, 2014).

In contrast to undercover political intelligence gathering, infiltration of organised crime networks by undercover officers attracts more public support. However, here too, innovative policing tactics have raised ethical dilemmas. In the next section, we discuss Operation Trojan Shield, a recent international policing operation where criminal organisations were surveilled via ΛNØM, an apparently encrypted device which, unbeknownst to the criminals, secretly sent copies of messages to the police.

ANOM

In 2018, a secure messaging company called Phantom Secure was suspended and shut down as the CEO Vincent Ramos was arrested in Washington. The Canadian company had provided many international criminals, such as high-level drug traffickers and other organised crime groups, with modified secure mobile phones (Federal Bureau of Investigation, 2018). An investigation revealed that they sold the devices exclusively to members of criminal organisations, particularly targeting transnational criminal organisations (Cheviron, 2021). Ramos was asked by the FBI to insert a backdoor into the device so that the criminal communications could be surveilled but he refused. However, with the closure of Phantom Secure, organised crime networks needed secure communications and the fact that its clientele seemed to consist only of criminals meant that law enforcement agencies, assessing that the general public would not be affected, saw an ideal opportunity to target criminal networks.

An international collaboration developed between the San Diego FBI office’s Operation Trojan Shield and the Australian Federal Police’s (AFP) Operation Ironside to develop a next generation encrypted device and app known as ANOM (often styled as AN0M or ΛNØM).

The FBI worked with a ‘Confidential Human Source’ (Cheviron, 2020) to develop and distribute the devices in exchange for a reduced sentence, $120,000, and travel expenses of around $60,000. Initially ANOM was beta tested with 50 users in Australia. Cheviron suggests that this trial was a success and enabled the AFP to penetrate two of the most sophisticated criminal networks operating in Australia. Importantly, for the project’s ethical viability, he adds that ‘according to Australian law enforcement, 100% of Anom users in the test phase used Anom to engage in criminal activity’ (Cheviron, 2020, p. 8). In other words, the technology was not being used for anything outside of crime. The operation moved on to the next phase and, by May 2021 there were about 9,000 devices in use. The devices – costing approximately £2,000 for a six-month service plan – sent and received encrypted electronic communications and stored data in encrypted form but had limited functionality. For example, users could not make normal phone calls or surf the internet. However, users were not aware that a master key was built into the device which surreptitiously attached to each message, allowing them to be instantly stored and decrypted by law enforcement. It was widely reported that over 800 people were arrested around the world, $48m in cash and cryptocurrencies and over 32 tonnes of drugs were seized, and more than 100 murder plots were counteracted. Europol reported that over 27 million messages were collected and it is expected that there will be further arrests in the future.

Privacy advocates have welcomed the fact that the operation did not involve inserting backdoors into products used by the general public, but they have also raised concerns. For example, Ashkan Soltani, previously the Chief Technologist of the Federal Trade Commission in the Division of Privacy and Identity Protection, stated that the operation showed that ‘You can use good old-fashioned detective work and operations without backdooring protocols and services that consumers widely use’ (Murphy, 2021). However, he went on to question the potential for the surveillance of innocent people. How many ‘non-targets’, he asked, were ‘swept up in this operation?’ (Murphy, 2021). Cheviron (2020) states that he believes ‘that Anom devices are used exclusively to openly discuss criminal schemes or to maintain relationships in furtherance of those schemes’ (p. 11). Presently, we can only assume that ‘non-targets’ were not caught up within the surveillance operation.

Concerns have also been raised about the impact of such operations on the legitimate encryption industry and about the way in which international law enforcement collaborations can enable national laws to be circumvented.

‘Laundering’ Surveillance

Jennifer Lynch, the Surveillance Litigation Director at the Electronic Frontier Foundation, has stated that US law enforcement was not able to monitor domestic Anom users because this would violate the Fourth Amendment and the Wiretap Act. Therefore, the USA relied upon other countries without these regulations ‘to launder its surveillance’ (Murphy, 2021). To circumvent US laws, the devices routed BCC encryptions of the messages to an iBot server outside of the USA, where it was decrypted, then re-encrypted with an FBI encryption code before being decrypted again for viewing (Cheviron, 2020). Around the middle of 2019, the investigators sought a third country to obtain an iBot server of its own because, although Australia’s judicial order allowed for the interception of Anom communications, it was unauthorised to share the information with foreign partners (Cheviron, 2020, p. 8, footnote 6). The mass raids and arrests took place on 8 June 2021, the day after the expiration of the court order allowing the third country to supply Anom server data to the FBI and this was probably no coincidence.

Greg Barns SC from the Australian Lawyers Alliance suggested that Australia was likely chosen as a partner in the operation because of its ‘very weak privacy protections’. He went on to state:

Often with these operations you go to the country with the weakest laws, as it were, so that you can obtain more evidence more easily and run less of a risk of evidence being obtained illegally. (Swanston, 2021)

Barns has argued that this is a form of entrapment wherein people are induced into committing a crime – entrapment is allowed in Australia but not in the USA.

Varying legal regimes mean that such international law enforcement collaborations provide potential societal benefits in terms of increased flexibility in mounting operations against well-funded targets but potential societal harms by undermining legal protections within each jurisdiction. This international operation has also reignited the debate about the legitimacy of public access to encryption.

The Rights and Wrongs of Encryption and Decryption

The US Department of Justice has made it clear that a goal of the operation was to target encryption. Randy Grossman, the acting US attorney said:

Hardened encryption devices usually provide an impenetrable shield against law enforcement surveillance detection. The supreme irony here is that the very devices that these criminals were using to hide from law enforcement were actually beacons for law enforcement. We aim to shatter any confidence in the hardened encrypted device industry with our indictment and announcement that this platform was run by the FBI. (United States Department of Justice, 2021)

Wired reports that the US Department of Justice and other law enforcement agencies have long lobbied for access to ‘end-to-end’ encrypted data from, for example, social media and other communication platforms (Newman, 2021). Since data are kept scrambled by companies so that they remain undecipherable along their journey across the internet, law enforcement agencies do not have access to their content, a problem they refer to as ‘going dark’. However, Wired argue that the FBI and, of course other agencies, have had continued success in finding creative ways of developing workarounds by, for example, targeting the devices rather than the encryption protocols themselves.

Some might argue that, given the success of operations like this, backdoors should be built into all apps. For example, in 2019 the UK’s Government Communications Headquarters (GCHQ) proposed that communication systems should be designed to include a silent, unseen participant like another member of the group chat, enabling government agencies to access them. However, there was a storm of reaction against this, not only from human rights groups but also from the Big Tech companies. Indeed, many of these companies introduced end-to-end encryption in the first place because of public reaction to the activities of the US and UK governments. Edward Snowden disclosed that, under the NSA’s PRISM programme, technology companies passed internet data to the NSA and that, under the MUSCULAR programme, GCHQ and the NSA had hacked into the main communications links connecting the data centres run by Yahoo! and Google without their knowledge. PRISM threatened public trust in technology companies whereas MUSCULAR threatened the companies’ trust in the US and UK governments. End-to-end encryption appeared to them to provide a solution to both problems.

Research suggests that, whilst there is wide public support for overt surveillance like CCTV, there is less support for covert and digital surveillance. The 34th British Social Attitudes (BSA) Survey reported that, although 80% of the public supported the use of video surveillance in public areas, 60% supported the collection of ‘information about anyone living in Britain without their knowledge’ and only 50% supported the monitoring of emails and other internet activity (Clery, Curtice, & Harding, 2016). Although Operation Trojan Shield will in the future be seen as a very successful method of counteracting serious organised crime, it will also serve to remind us that our online lives are always in danger of being covertly surveilled.

Operation Trojan Shield threw up some unique challenges. For example, the FBI needed to ensure both that the general public was not affected and that the fake encrypted phone company’s cover was maintained. Andrew Young, a partner in the Litigation Department in law firm Barnes and Thornburg stated ‘We can’t just run a good investigation; we have to run a good company’ (Cox, 2021). This included ensuring both that the marketing of the company was done correctly, and that the fake company was credible. In order to gain and maintain good customer service and satisfaction they had to provide technical support and deal with hackers. Importantly, they had to make sure that it did not become mainstream – they could not allow it to get into the hands of the public because of the ethical issues related to surveilling non-targets. Hence, distribution needed to happen within the criminal circles. A key unwitting distributor was Hakan Ayik who had long standing connections with Australian biker gangs and was an alleged drug lord. Ayik is currently an international fugitive, wanted not only by the authorities but, presumably, by previous customers who hold him responsible for their predicament (BBC News online, 2021).

Operation Trojan Shield is a good example of surveillance through data and digital technologies. Indeed, the capture, processing and categorisation of data has unsurprisingly become a significant part of surveillance studies and raises significant ethical challenges (Harper et al., 2013; Tucker, 2013; Van Dijck, 2014). Another key area, outside of law enforcement, in which data capture and processing is a growing concern is in relation to children’s learning in schools, and the associated role of forms of education technology (so-called ‘EdTech’). The use of digital learning platforms, and associated technologies in schools, has risen significantly during the Covid-19 pandemic. However, it is not clear that governance structures have kept pace with their increased use, or with the new technological developments on the horizon (e.g. use of artificial intelligence, AI). The next section focusses on some of the ethical concerns of EdTech, with a specific focus on the UK context.

Governance of Education Technology in Schools

Children are, by definition, classed as a vulnerable group, and yet there is significant opacity regarding the governance of the use of education-focussed technologies such as Google Classroom in UK schools. This point is a key message from a recent Digital Futures Commission report (Day, 2021), which undertook a detailed analysis of the data-related legislation, and associated governance processes (at government and school level), in relation to the use of what they refer to as ‘EdTech’. Whilst there are clear legislative frameworks for data processing, such as GDPR, there is no specific legislation focussing on the use of EdTech in schools, which given that its use, and therefore the role of the private sector, has increased significantly in recent years, is somewhat of a surprise. This lack of a legislative framework creates a governance vacuum, as schools and local education authorities (LEAs) do not have clear legislation upon which to develop and implement their local governance practices. The current system also places significant responsibility on schools to manage governance, as policies allow and encourage schools to identify their own EdTech systems, meaning that differ-ent schools can use different platforms (although the ‘free to use’ policies of big players such as Google Classroom means that certain platforms are coming to dominate).

The fact that legislation lags behind the data generating and processing practices of EdTech makes it difficult to identify the entirety of the ethical concerns in relation to children’s data in schools. With children having to attend school by law (unless they have a home-schooling agreement with their LEA), they have no choice but to engage with any EdTech used by their school. This makes the use of EdTech such an important ethical issue, because children cannot avoid it. The opacity regarding the governance of data processing activity means that forms of covert surveillance emerge. For instance, does a child understand that if they opt-in to an associated product/service provided by their EdTech provider, they could be consenting to the company to use their data for marketing purposes – and that such activity involves a direct contract between child and digital platform, outside of any school policy (Day, 2021)?

AI and EdTech

Concerns about the potential for surveillance of children’s learning are broadening in relation to new developments involving the use of forms of AI in digital learning platforms. This step potentially signifies a move towards automated forms of learning, whereby children can ask an AI-driven conversational agent questions related to learning. Google CEO, Sundar Pichai, recently announced its foray in this area, an AI-driven system called LaMDA (Language Model for Dialogue Applications), which is a natural language processor-based conversational agent that children can ask questions of and subsequently receive responses in a conversational format (Williamson, 2021). We know that such systems rely on ‘learning’ from the data gathered from previous interactions, so will involve mass aggregation of data related to children’s learning, and as such, involve widespread surveillance of engagement with digital learning platforms. Given the fact that AI and data mining are dependent on having large amounts of data, there is significant incentive to expand aggressively into new domains like education. In addition to major technologies and data companies such as Google moving into large scale data mining in education, there are also smaller technology firms drawn to education and the development of AI-driven tools to capture and categorise children’s learning.

One example is the use of AI-based emotion detection systems, for example, http://www.4littletrees.com/ which has been used in secondary education in Hong Kong (Murgia, 2021). 4 Little Trees is an AI-driven system that is designed to identify and monitor children’s emotional responses and activity during online lessons. The aim is to provide feedback to teacher and schools about when students lose attention, and whether this informs as to the effectiveness of teaching practices and allow for teachers to respond to children’s learning in ‘real time’: for example, if 4 Little Trees suggests a child is losing attention a teacher can ask a question to that child to re-engage them. 4 Little Trees is based on facial recognition systems that have been used by law enforcement and border control agencies in recent years. 4 Little Trees extends the ‘recognition’ capabilities of such systems through claiming to be able to identify not the person’s identity, but their emotions, feelings, sentiments. The growth of emotion-related facial recognition systems, which have been named ‘emotional-AI’ has been significant in recent years (McStay, 2018, 2020). The attraction to advertisers of being able to identify individuals’ emotional responses to adverts is a major one and is driving the industry. Its use in education is at the embryonic stage, but there is no reason to think that education will be naturally immune to the desire and push to automate that AI-driven systems offer.

There are important points to note regarding emotion-AI systems such as 4 Little Trees. Firstly, that the data collected from children are of an intrusive nature (e.g. emotional state, videos of children in their homes and so on). Secondly, it is not entirely transparent how the data will be used, both by the private sector (i.e. 4 Little Tress) and by schools. Whilst 4 Little Trees states that data collected by authority figures (i.e. teachers) will be used to make decisions about children’s engagement with learning, it is not clear how it might be used in the future. Finally, the universal model of emotion that such technologies are developed from (i.e. that a core set of emotions exist with largely universal modes of expression) has been extensively critiqued (Barrett, 2018; Barrett, Adolphs, Marsella, Martinez, & Pollak, 2019; Ellis & Tucker, 2020). The implication of these critiques is that the categorisation of the emotional states of children cannot reliably be taken as accurate. If we cannot rely on interpretations of systems such as 4 Little Trees, it is problematic to base elements of children’s education on them. An expression of inattention could relate to a child reflecting on a problem relating to their learning, rather than inattention per se. Furthermore, if a child is deemed to be inattentive and unengaged, despite previous warnings, would this lead to punishments?

The real time monitoring, tracking and categorising of children’s facial expressions during online learning is an example of an emerging form of surveillance. Whilst it resonates with traditional notions of top-down power, in the form of powerful organisations (such as technology companies and schools) initiating and undertaking the surveillance, its operation is closer to what Isin and Ruppert (2020) refer to as sensory power, which involves ‘data that tracks and traces people in their movements, sentiments, needs and desires’ (p. 2). In the case of 4 Little Trees, it is the tracking and categorising of facial expressions in terms of emotion, mood and attention. Isin and Ruppert claim sensory power is a new form of power that is distinguished from traditional notions of sovereign, disciplinary and regulatory power. Data tracking technologies such as emotion-AI systems have made possible more sophisticated forms of surveillance in terms of focussing down on specific psycho-physiological activity, such as with micro-facial expressions. Others forms of tracking have also emerged, such as fitness trackers that can capture and categorise heart rate, skin conductance and so on.

The ethical challenges of EdTech are only going to continue to grow as with increased use of AI in digital learning platforms. To date, much of the data under focus has involved things such as children’s IP addresses, time spent engaging with platforms, wider patterns of use and such like. The advent of tools such as 4 Little Trees adds an additional layer because it generates different kinds of data about children. The processing of descriptive data about patterns of use (such as location, duration) is added to data interpreting and categorising children’s faces directly in relation to emotion and attentional state. This is a more sophisticated level of data, which is seen as attractive due to its potential to inform regarding the effectiveness of different forms of online learning. However, its categorising of facial expressions as informing of emotional states, based on problematic emotion science, makes it both intrusive and potentially inaccurate due to not being scientifically valid.

Allowing private companies to develop and use facial recognition technologies in children’s learning environments presents major ethical challenges, from concerns regarding data protection through to allowing private companies access to videos of children’s engagement in ‘real time’ learning in their homes. This is an emerging area of concern, for which new forms of governance are required. Whilst the 4 Little Trees system is not currently in use in the UK, it is indicative of one direction that EdTech is taking, and as such it is an important example of the considerations for governance processes. As the Digital Futures Commission Report notes:

[G]iven the lack of data governance or data analytics expertise in schools, putting the responsibility on schools to negotiate these contracts puts a large amount of power in the hands of EdTech companies to interpret and apply data protection laws in a way that suits their own commercial purposes, without any oversight. (Day, 2021, p. 46)

In relation to the current use of EdTech in UK schools, the system’s positioning of responsibility at the level of schools, which can often lack the detailed technical knowledge to map sophisticated data generating and processing practices onto existing governance, is not an optimal strategy to ensure transparent and ethical governance structures. Legislation and governance structures are required to be developed and implemented at the level of government and given the significant increase in the use of EdTech during the Covid-19 pandemic, the need is significant and pressing (Day, 2021).